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Abstract — Many existing complex space systems have a sig- 
nificant amount of historical maintenance and problem data 
bases that are stored in unstructured text forms. For some 
platforms, these reports may be encoded as scanned images 
rather than even searchable text. The problem that we address 
in this paper is the discovery of recurring anomalies and rela- 
tionships between different problem reports that may indicate 
larger systemic problems. We will illustrate our techniques on 
data from discrepancy reports regarding software anomalies 
in the Space Shuttle. These free text reports are written by 
a number of different people, thus the emphasis and wording 
varies considerably. 

We test three automatic methods of anomaly detection in text 
which are popular in the current literature on text mining. The 
first method that we describe is k-means or Gaussian mixture 
model and its application to the term-document matrix. The 
third method is based on an analysis of the results of apply- 
ing a new clustering method, based on the von Mises Fisher 
distribution, that represents each document as a point on a 
high dimensional sphere. In this space, we perform cluster- 
ing to obtain a set of potential anomalies. We also describe 
results that are derived from a new- method known as spectral 
clustering, where vectors from the term-document matrix are 
embedded in a high dimensional space for clustering. 

The paper concludes with recommendations regarding the de- 
velopment of an operational text mining system for analysis 
of problem reports that arise from complex space systems. 
We also contrast such systems with general purpose text min- 
ing systems, illustrating the areas in which this system needs 
to be specified for the space domain. 
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1. INTRODUCTION 

Many complex aerospace systems have a variety' of prognos- 
tic and diagnostic instrumentation that deliver high speed data 
streams of information regarding the current health of the sys- 
tem. These streams give instantaneous information about the 
system and must be analyzed accordingly. 

Along wtith these data streams, however, aerospace systems 
also have significant maintenance records associated with 
them. These maintenance records are often free text reports. 
They are often recorded by maintenance personnel or engi- 
neers that are responsible for specific subsystems in the ve- 
hicle. In some cases, such as the Aviation Safety Report- 
ing System [1], the reports are augmented by some structured 
data through the use of a coded report. The coded reports can 
be analyzed using standard statistical methods or data mining 
methods that are suited for the analysis of structured informa- 
tion. 

The free text reports, however, need to be significantly trans- 
formed to be analyzed wtith standard data raining or statistical 
methods. Most of those methods assume that the data can be 
expressed as a matrix where each row is an observation and 
each column is a variable. For example, in the case of ana- 
lyzing the variations in reliability for 1000 different thermal 
sensors, a matrix could be formed which would have 1000 
rows and columns corresponding to various reliability met- 
rics as well as other information regarding the sensors that 
are deemed relevant by the analyst. This information could 
include, for example, where the sensor was manufactured, 
when it was manufactured, information regarding the man- 
ufacturing process, etc. These pieces of information would 
form the columns of the data matrix that could then be sub- 
mitted to a statistical or data mining analysis. 

This paper discusses methods of analyzing free text docu- 
ments where the text is represented in a matrix as described 
above- each document corresponds to a row in the matrix, 
and the columns correspond to the union of all the key words 
in all the documents. The entries in the matrix (called a term- 
document matrix) correspond to the frequencies of each key 
word (or term) in the document. Through this procedure each 
document is represented in a point in a high dimensional vec- 
tor space. This representation is used by many text analy- 
sis methods under the terms ’bag-of- words’, latent semantic 
analysis, and other research areas [2]. A significant drawback 
of this vector space approach is that all semantic and syntactic 
information in the document is lost. 

In the next section, we motivate the particular problem that 


we use to demonstrate our methodology, which is detecting 
recurring anomalies in free text repons that are used in the 
Space Shuttle’s Flight Readiness Review's. In Section III, we 
describe two methodologies that are standard for analyzing 
text documents that are represented in vector spaces. In Sec- 
tion IV, we briefly review the relevant algorithms. In Section 
V we discuss our experimental results and in Section VI we 
conclude the paper by summarizing it and discussing future 
work. 

2. Discovering Recurring Anomalies 

The problem that we address in this paper is as follows. Given 
a set of N documents, where each document is a free text 
English document that describes a problem, an observation, a 
treatment, a study, or some other aspect of the the vehicle, au- 
tomatically identify a set for potential recurring anomalies in 
the reports. Note that for many applications, N « 100. 000, 
which is a corpus that is too large for a single person to read, 
understand, and analyze by hand. Thus, w'hile engineers and 
technicians can and do read and analyze all documents that 
are relevant to their specific subsystem, it is possible that 
other documents, which are not directly related to their sub- 
system still discuss problems in the subsystem. While these 
issues could be addressed to some degree with the addition of 
structured data, it is unlikely that all such relationships would 
be captured in the structured data. Therefore, we need to de- 
velop methods to uncover recurring anomalies that may be 
buried in these large text stores. 

One approach to addressing this problem would be to develop 
a method to query the text database for known anomalies. For 
example, one could envision generating a list of queries, such 
as "find all examples of software errors", or ’’find all exam- 
ples of navigation system faults”, etc. While such a query 
mechanism is useful, it still does not address the problem of 
finding anomalies that may not be thought of a priori. The ap- 
proaches that we describe in this paper are particularly useful 
for identifying unknown recurring anomalies. 

The methods that we use to discover these anomalies are 
based on various clustering methods. Clustering refers to the 
process of identifying subsets of rows in the term-document 
matrix that have similar characteristics. The first approach 
that we discuss is based on the k-means clustering algorithm 
of the term-document matrix which implicitly makes Gaus- 
sian assumptions and uses the Euclidean distance between 
term-document vectors as a measure of similarity. The sec- 
ond clustering method uses the cosine measurement between 
two vectors and which implicitly assumes the von Mises 
Fisher distribution. The third clustering method, based on 
spectral clustering, embeds the term-document vectors in an 
infinite dimensional space and looks at the clustering of a low 
dimensional projection. These formulations will be discussed 
in the next section. 

Our procedure to identify recurring anomalies is based on the 
idea that they will show up in the same cluster, and thus is 


hishlv dependent on the clustering algorithm. In this section, 
we describe three methods of cluster analysis that are popular 
in the literature today and discuss their underlying assump- 
tions. These assumptions affect the outcome of the clustering 
and therefore can affect the discovery of recurring anomalies. 

For purposes of the discussion presented here, we will model 
the text as a term-document matrix [3]. The term-document 
is described by an N x p matrix Z, where N is the number 
of documents, and p is the number of keywords in the union 
of all documents. A keyword is defined as a word that is in- 
formative about the content of the document. Words such as 
’and’, ’the’, ’but’, and ’not’ are called stop words and are 
abandoned when the term-document matrix is created. In 
many applications, p ;» N. In order to remove terms from 
this matrix that have small frequencies as compared to the 
number of documents, it is customary to perform a data re- 
duction technique known as Term Frequency Inverse Docu- 
ment Frequency (TFIDF) to the term document matrix. We 
follow the notation in [3] as follows. For Z t j, which corre- 
sponds to the entry in the matrix for the tth document d- L and 
the jth term tj, TFIDF is a straightforward procedure and can 
be computed as follows: 

Zij = TF(tj,di) x IDF(tj) (1) 

TF{tj,di) is the term frequency, which is the frequency that 
term tj appears in document d*. IDF(tj) is the Inverse Doc- 
ument Frequency of term tj and is defined as: 

/DF«,) ^ogljUU.) (2) 

where DF{tj) is the number of times that term tj appears 
in the corpus. Notice that if this number is close to N, the 
number of documents in the corpus, IDF(tj) ~ 0, and the 
term’s contribution to the matrix is very small. 

Dimensionality Reduction using Principal Components Anal- 
ysis 

Insert text here. 

k-means Algorithm and Mixture Models 

The k-means clustering algorithm [4] is perhaps the most 
popular method of clustering structured data due to its sim- 
plicity of implementation. The algorithm works by choos- 
ing k random initial cluster centers, computing the distances 
between these cluster centers and each row in the data ma- 
trix and then identifying those rows that closest to each clus- 
ter center. The corresponding cluster centers are moved to 
the centroid of those data points and the procedure is re- 
peated. The algorithm converges when the cluster centers do 
not move from one iteration to the next. 

The k-means algorithm is a special implementation of the 
Gaussian Mixture Model. These models assume that the 
data vectors are generated according to the probability den- 
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sity P(Zj\Q) and assume that: 

G 

P(Zi\Q) = P(c)P(Zi\9 c) (3) 

c=i 

where © is a vector containing the C model parameters, 
and # c are the model parameters for the cth mixture com- 
ponent. The vector Z z is a p dimensional vector from the 
term-document matrix. The parameters of this model are ob- 
tained through Expectation Maximization of the appropriate 
log-likelihood function or, more generally, the posterior log- 
likelihood. In the case of a Gaussian mixture density model 
for Zi , we take the likelihood function as: 

P{Zi\9 c ) = P(Zi\ti c ,Z c ,c) 

= (27r)-f!E c |-i x 

expr^t -Mc) T S~ 1 (Z i -p c )) 

Maximum a posteriori estimation is performed by taking the 
log of the posterior likelihood of each data point Zi given the 
model © using the Expectation Maximization algorithm [5]. 

In the case of text clustering the vectors are high dimensional 
and sparse. In this case, the k-means algorithm does not 
work well because the number of data points needed to from 
dense regions increases exponentially with the number of di- 
mensions. With a finite amount of data and high dimension, 
most data points end up being approximately equidistant to 
each other. Figure 1 show's the effect of increasing the num- 
ber of dimensions on the average Euclidean distance between 
points. The x-axis is a logarithmic scale from 10 to 10,000 di- 
mensions. From top to bottom, the curves indicate the effect 
of sparseness of the vectors. The bottom curve corresponds to 
a vector that is 50% sparse, the remaining curves correspond 
to vectors that are 66%, 75%, and 90% sparse, respectively. 

Sammon Nonlinear Mappings 
Insert text here. 

von Mises Fisher Clustering 

The Gaussian Mixture Model and k-means algorithms make 
Gaussian assumptions about the underlying distribution of the 
data. Empirical studies have showm that for high dimensional 
sparse data sets, the cosine measure of similarity between two 
vectors is a better measure than the Euclidean distance. A re- 
cent paper [6] developed the mathematics to perform clus- 
tering using the cosine measure of similarity. Just as the 
Euclidean distance implicitly implies a Gaussian distribution 
the cosine distance implicitly implies a different distribution, 
known as the von Mises Fisher distribution. We follow the 
formulation in [6] closely: 

c 

p(z i \e) = '£p( C )P(z l \e c ) ( 4 ) 

C=1 

In this case, we assume that the vectors Z x have been normal- 
ized to unit length. For p dimensional data vectors, we have 
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Figure 1. The effect of sparseness and increasing number of 
dimensions on the Euclidean distance between points. Notice 
that for a given sparseness (corresponding to one curve) the 
average distance between points increases wdth the number of 
dimensions. 


the von Mises Fisher (vMF) distribution: 

P(Zi\p, k) = c p (k) exp {Kp T Zi) (5) 


where p is a unit vector corresponding to the mean of the 
distribution and k > 0 is the measure of dispersion. The 
constant Cp(n) is given by: 


pf K 1 — 
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( 6 ) 


where I T ) (k) represents the modified Bessel function of the 
first kind of order r. With the vMF distribution as defined 
above, Baneijee et. al. 2003 derive the Expectation Max- 
imization algorithm to optimize a mixture of vMF distribu- 
tions. Their results indicate that this algorithm has superior 
performance on high dimensional text clustering problems 
compared to the k-means algorithm. 


Spectral Clustering 

Spectral clustering is a different approach to clustering that 
works by embedding the vectors Z, in a high, possibly infinite 
dimensional space using Mercer Kernels [7]. Mercer Kernel 
functions can be viewed as a measure of the similarity. For 
a finite sample of data Z, the kernel function yields a sym- 
metric N x N positive definite matrix, where the (i.j) entry 
corresponds to the similarity between (Zi, Zj) as measured 
by the kernel function. Because of the positive definite prop- 
erty, such a Mercer Kernel can be written as the inner product 
of the data in the feature space. Thus, if $(Z X ) : TV *-* P 
is the (perhaps implicitly) defined embedding function, we 
have K(Zi,Zj) = $(Zi)$ T (Zj). Typical kernel func- 
tions include the Gaussian kernel for w'hich K(Zi,Zj) = 
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Figure 2. The top panel of this plot shows the sorted inter- 
document distances in the original 500 dimensional space and 
the distances that arise from a 2 dimensional approximation 
to the original distances. Original distances are shown in the 
solid line, and the dotted line shows the distances with the 
2 dimensional approximation. Notice that there is substan- 
tial error in the approximation. The middle panel shows the 
results of Sammon mapping after the dimension of the docu- 
ment space is reduced from 500 dimensions to 10 dimensions 
using principal components analysis. The agreement between 
the distances in the low dimensional space and the 2 dimen- 
sional mapping are excellent. The bottom panel shows the 
approximation of the Sammon mapping using a neural net- 
work. 


§(Zi)$ T (Zj) = exp(- 2^j\\Zi - Zj\\ 2 ), and the polyno- 
mial kernel K (Zj, Zj) = $(Zi)$ T (Zj) =< Zi,Zj > p . 

For supervised learning tasks, linear algorithms are used to 
define relationships between the target variable and the em- 
bedded features [8]. Work has also been done in using kernel 
methods for unsupervised learning tasks, such as kernel clus- 
tering [9], [?] and density estimation [10]. 

Spectral clustering works by computing the eigenvectors of a 
normalized kernel matrix (see [7] for details of the algorithm). 
The largest n eigenvectors are chosen and normalized to unit 
length. The rows of the eigenvectors (corresponding to N 
points in an n dimensional space) are then clustered using the 
k-means algorithm. 

In Table 1, we show the asymptotic efficiency of the algo- 
rithm. 

3. Developing an Operational Text 
Mining System 

In this section we describe a system architecture for an oper- 
ational text mining system. An aerospace vehicle is a highly 



Figure 3. This visualization is a projection of the 1 25 dimen- 
sional document vectors into two dimensions using Sammon 
mapping. The contours represent regions of equiprobability. 
Recurring anomalies can be documents that fall within the 
same closed contour. 
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Figure 4. This visualization shows how the clustering re- 
sults vary with the three parameters in spectral clustering: the 
number of dimensions, the number of clusters, and a 2 , which 
is the scale parameter in the kernel. 


complex system with complex interactions between its vari- 
ous subsystems. To get the most out of a problem tracking 
system as many of these complex. relationships as possible 
need to be included in the tracking and analysis of issues that 
arise. The system we propose contains an engineering model 
of the vehicle detailing the relationship between vehicle com- 
ponents and subsystems. This model is joined to a relational 
database containing additional vehicle component informa- 
tion as well as structured fields for entering problem reports. 
This information gives the system a better context in which 
to do clustering of the problem reports, thereby increasing 
the likelihood that meaningful clusters will be produced. The 
need for this type of organized, interconnected structure was 
found to be important in the NASA Space Shuttle program by 







Figure 5. This visualization is a projection of the 500 dimen- 
sional document vectors into two dimensions using Sammon 
mapping. The contours represent regions of equiprobability. 
Recurring anomalies can be documents that fall within the 
same closed contour. 



Figure 6. This diagram shows the frequency distributions of 
documents with clusters. The large cluster corresponds to the 
dense region in the previous figure. 


an independent assessment team [11]. 

The vehicle model should consist of ontology of the language 
used to describe the vehicle and its components, domain in- 
formation, and vehicle system structure. 

The ontology portion defines the language of terms used 
when describing problems with the vehicle. This includes 
acronym definitions, thesaurus terms, conceptual hierarchies, 
and irrelevant terms. Commonly used acronyms need to be 
defined so that their terms can be related between documents 
with related, but not identical, references. A set of thesaurus 
terms will help to relate documents by their intended mean- 
ing, not just their literal content. Conceptual hierarchies 
group sets of terms into low level concepts, and low level con- 


Table 1. Asvmptotic Efficiency of the Detection Algorithm 
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Figure 7. System Architecture - Engineers observing the ve- 
hicle enter problem reports into the system, which are stored 
in a relational database and joined with vehicle component 
and design information. This allows for flexible reporting and 
more relevant analysis of trends in the problem reports. 


cepts into higher level concepts. It can be thought of as a tree 
structure, with all of the terms in the language at the leaves 
of the tree. The parent node of a set of terms is the concept 
shared by all of those terms. At the next level up the tree these 
low level concepts are joined by a parent node which groups 
them into a higher level concept. This continues up the tree 
(hierarchy) to the root node, which joins all the highest level 
concepts together and represents the base concept of the en- 
tire language. This hierarchical structure helps to put terms in 
context and to create links between documents. The weight 
of the links can be varied depending on the level in the con- 
ceptual hierarchy that the link was made. A set of irrelevant 
terms should also be included in the vehicle model for com- 
mon terms or codes that shouldn’t be used when clustering 
documents. 

The domain information consists of relationships between 
terms. Terms can be related by causality (ie. ’water’ causes 
’corrosion’), similarity, mutual exclusivity, etc. These rela- 
tionships should describe physical and engineering relation- 
ships that are specific to the vehicle design. 

The vehicle system structure is an engineering model that de- 
fines how parts, components, and subsystems interact with 
each other. 

The relational database consists of tables for all of the vehicle 
parts, components, and subsystems. It also has transactional 
tables for entering problem reports with both fixed fields and 
free text fields. Setting up the database for problem tracking 
in this manner will allow' for simple and complex queries to 
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Figure 8. Example vehicle system structure - specifies 
how components and subsystems fit together so that analysis 
methods can take into account interactions between subsys- 
tems. 


answer common high level questions as well as give a great 
deal more information to the clustering algorithms. 

The part table should have a part ID as a primary key. Each 
record should be a unique part with fields describing proper- 
ties of the part as well as component IDs of each component 
the part is used for in the vehicle. These can be used for join- 
ing with the component table. 

Similarly, the component table should have a component ID 
as the primary key and each record should describe proper- 
ties of the component. It should list subsystem IDs of each 
subsystem the component is part of. 

The problem report tables are similar to bug tracking systems 
commonly used in software development. They should con- 
sist of fixed fields for things like title, priority, problem cate- 
gory, severity, and subsystem or component if applicable. The 
free text field is the main body of the problem report where a 
full description and discussion of the problem is entered. 



Figure 9. Relational Database Schema - joins problem 
reports with detailed information about the vehicle’s parts, 
components, and subsystems. 

The system can be used to perform clustering of problem re- 
ports in order to discover recurring anomalies. As observers 
discover problems on the vehicle they enter them into the re- 
lational database through a simple web interface. If the ve- 
hicle system design changes then the vehicle model is up- 


dated to reflect those changes. Regular reports of open is- 
sues can be generated simply by querying the database. A 
streamlined, efficient process flow for entering and analyzing 
problem reports is critical for analyzing trends and recurring 
anomalies [12]. To this end, the system architecture, cluster- 
ing algorithms, and methods described above can be used. 

4. Data Analysis 

These are the steps we took to create clusters from a set of 
Flight Readiness Reports and Discrepancy Reports for the 
space shuttle. We first converted the documents into plain 
text format. Some documents were scanned images of printed 
pages. For these we performed OCR (Optical Character 
Recognition) to convert the images to text. 

Since we had a small number of documents, and there were 
repeating sections throughout the documents, we broke the 
documents apart into sections which were treated as indepen- 
dent from one another. 

We next created a Bag of Words matrix. This matrix has a 
column for each keyword appearing in the entire collection 
of documents and a row for each document. For each row 
vector, which represents a single document, the number of 
occurrences of each keyword is placed in the appropriate col- 
umn of the matrix. 

In order to weight distinctive terms properly during the clus- 
tering we then applied tfidf. 

For some algorithms the dimensionality (number of columns) 
of the bag of words matrix was too high. We reduced the 
dimensionality by performing Singular Value Decomposition 
and selecting just the fist 30 dimensions. 

Each document vector was normalized by dividing each ele- 
ment by the vector’s L2 norm. 

We were then able to apply the clustering methods described 
above. 

5. Conclusions 

For the Discrepancy Reports we also were given a set of 
groupings that was done by shuttle software team members. 
We were able to compare our own clustering results with 
these. We found that in several cases, documents that were 
very similar had been grouped in separate clusters by the soft- 
ware team members but were identified with the same cluster 
by our clustering software. 

6. ACKNOWLEDGEMENTS 

The authors thank Dr. X for useful discussions. The research 
was supported in part by the U.S. ONR grants #. 




References 

[1] L. Connell, “Incident reporting: The nasa aviation 

safety reporting system," GSE Today , pp. 66-68, 1999. 

[2] T. K. Landauer, D. Laham, and P. Foltz, “Learning 

human-like knowledge by singular value decompo- 
sition: A progress report,” in Advances in Neu- 

ral Information Processing Systems, M. I. Jordan, 
M. J. Kearns, and S. A. Solla, Eds., vol. 10. 
The MIT Press, 1998. [Online]. Available: cite- 
seer.ist.psu.edu/Iandauer981earning.html 

[3] T. Joachims, “A probabilistic analysis of the Roc- 
chio algorithm with TFIDF for text categorization," 
in Proceedings of ICML-97, 14th International Con- 
ference on Machine Learning , D. H. Fisher, Ed. 
Nashville, US: Morgan Kaufmann Publishers, San 
Francisco, US, 1997, pp. 143-151. [Online]. Available: 
citeseer.ist.psu.edu/joachims96probabilistic.html 

[4] M. I. Jordan and R. A. Jacobs, “Hierarchical mix- 

tures of experts and the EM algorithm. Tech. 
Rep. AIM- 1440, 1993. [Online]. Available: cite- 

seer.ist.psu.edu/article/jordan94hierarchical.htmJ 

[5] A. P. Dempster, M. Laird, N., and D. B. Rubin, “Maxi- 
mum likelihood from incomplete data via the em algo- 
rithm,” Journal of the Royal Statistical Society B , 1977. 

[6] A. Baneijee, I. Dhillon, J. Ghosh; and 

S. Sra, “Generative model-based clustering of di- 
rectional data,” 2003. [Online]. Available: cite- 

seer.ist.psu.edu/article/baneijee03generative.html 

[7] A. Ng, M. Jordan, and Y. Weiss, “On spectral 
clustering: Analysis and an algorithm,” 2001. [Online]. 
.-Mailable: citeseer.ist.psu.edu/ng01spectral.html 

[8] N. Cristianini and J. Shawe-Taylor, An Introduction to 
Support Vector Machines. Cambridge University Press, 
2000 . 

[9] M. Girolami, “Mercer kernel based clustering in feature 
space,” IEEE Transactions on Neural Networks, vol. 13, 
no. 4, pp. 780-784, 2001. 

[10] W. G. Macready, “Density estimation with mercer ker- 
nels,” Technical Report TR03.13 of the Research Insti- 
tute of Advanced Computer Science, 2003. 

[11] H. McDonald, “Shuttle independent assessment team 
report,” Splace Shuttle Independent Assessment 
Team Report to Associate Administrator Office of 
Space Flight, Tech. Rep., 1999. [Online]. Available: 
www.hq.nasa.gov/osf/siat.pdf 

[12] C. Linde and R. Wales, “Work process issues 
in nasa’s problem reporting and corrective action 
(praca) database,” NASA .Ames Research Center, 
Human Factors Division, Tech. Rep., 2001. [On- 
line], Available: human-factors.arc.nasa.gov/april01- 
workshop/2pg-linde3.doc 


Ashok N. Srivastava is a Principal Sci- 
entist and Group Leader in the Data 
Mining and Complex Adaptive Systems 
Group at NASA Ames Research Center. 
He has fourteen years of research, de- 
velopment, and consulting experience in 
machine learning, data mining, and data 
analysis in time series analysis, signal 
processing, and applied physics. Dr. 
Srivastava has had significant experience both in research 
(NASA, NIST, IBM) as well as the business world at IBM 
(Senior Consultant ) and Blue Martini Software (Senior Di- 
rector). 

Dr. Srivastava 's machine learning research interests include 
topics in kernel methods, assessment of linear and nonlin- 
ear covariability, understanding and forecasting time-based 
data, and image processing. He is also interested in dis- 
tributed data mining and scalability issues in federated data 
systems. A primary area of applied research is in the develop- 
ment of onboard satellite algorithms for automatic detecting 
and discovery of geophysical processes. 


Brett Zane-Ulman is a Computer Scien- 
tist at Computer Sciences Corporation 
(CSC), in the Data Mining and Complex 
Adaptive Systems Group at NASA Ames 
Research Center. He has seven years of 
software development and consulting ex- 
perience in data mining, and data visu- 
alization. He has developed enterprise 
software at SGI and Blue Martini Soft- 
ware. 





